1. Field of the Invention
The present invention relates to image processing in general, and to a method and apparatus for stabilizing images in video in particular.
2. Discussion of the Related Art
Digital cameras, especially digital video cameras, may move while capturing an image. Movement of cameras may include voluntary as well as non-voluntary movements. Non-voluntary movements may be caused by non-voluntary movements of the person holding the camera, by movement of a tripod on which the camera is mounted, or by other factors, such as wind. As a result of the movements, when the camera moves, objects in images may be distorted, dislocated or otherwise incorrectly imaged. Another result is blurred images, in which the details are hardly recognizable. Yet another result is objects in the scene appearing both in the correct location and in misplaced location determined by motion compensation, or objects appearing distorted.
Two popular types of sensors for capturing images are used in digital cameras. One is CCD sensor that enables capturing the entire image simultaneously, and the other is CMOS sensor in which the image is captured in parts. For example, the image may be captured row-wise or column-wise, wherein the rows or columns are captured sequentially. Thus, when using CMOS sensors, if the camera moves while capturing the image, objects or parts thereof may be dislocated. Consequently, one row may require more compensation than a previously captured row. In addition to movement of the camera, movement of objects during exposure of a portion of the image also requires motion compensation when using CMOS sensors.
Referring to FIGS. 1A, 1B and 1C, schematically illustrating three images taken by a moving camera. FIG. 1A was captured using a CCD sensor, while FIGS. 1B and 1C were captured using a CMOS sensor. All three images show a tree perpendicular to the ground. The lower portion of the images comprises two lines, one is the base line (104 in FIG. 1A) and the other is the trunk (102 in FIG. 1A). The upper portion comprises the treetop of the tree (106 in FIG. 1A). On FIG. 1A, camera movement causes movement of all parts of the tree, since the entire image is captured simultaneously. Line 104 is perpendicular to line 102, but the entire tree appears at the left side of the image.
The image in FIG. 1B was captured using CMOS sensor while the camera was moving slowly. Hence, objects are located more on the right-hand side of the image, instead of being located in the center of the image. Rows in the image are captured at different times. For example, if the scanning direction is top-to-bottom, i.e. a top row is scanned before a bottom row, and then lower parts of the image in FIG. 1B are captured after the upper parts. The slow movement causes the lower parts of the image to be slightly moved to the right hand side of the image. Further, the difference between the times each row is captured causes the lower portion of line 104 to be more shifted to the right than the upper portion of line 104. As a result, base line 112 is not perpendicular to trunk 114 in FIG. 1B.
In FIG. 1C, fast movement of the camera causes significant differences in the location of objects in the image. The motion compensation required in FIG. 1C is more significant than the compensation required in FIG. 1B, since the difference between the correct locations of objects and the captured locations is larger than the differences between FIG. 1B and FIG. 1A. For example, base line 122 in FIG. 1C is located on the extremity of the right portion, while base line 112 in FIG. 1B is located in the middle right portion. Further, treetop 126 of the tree shown in FIG. 1C is distorted relatively to treetop 116 and treetop 126 in FIGS. 1A and 1B respectively, since the fast camera movement also causes significant change in the location of the upper objects in FIG. 1C.
One solution for video stabilization is to change the location or angle of an optical element, such as a lens or the camera sensor. The element is preferably displaced in the opposite direction to the camera movement direction. This solution is hardware based and requires a two-step method: first detecting the movement in terms of speed and direction and then calibrating the optical elements accordingly. The solution is relatively inaccurate and requires expensive and sensitive hardware elements.
Another solution for image stabilization is to compare a current image with to previously captured images.
The difference between a current image moved by a predetermined offset, and a previous image is determined for multiple offsets. The offset for which the difference is minimal is the preferred offset between the images and is used for correcting the image. The offset values are stored in a correlation matrix, in which the rows represent vertical offset, the columns represent horizontal offset, and the value within each entry in the matrix is the difference. The solution enables motion compensation but requires sufficient memory for storing images in their entirety. In video cameras, as in all digital cameras, memory is an expensive resource and it is desirable to provide a solution for image stabilization that provides real time motion compensation without requiring additional memory. Further, comparing images in their entirety requires significant processing power and consumes time that may disable real time stabilization of video or image.
There is thus a need for image stabilization techniques, which can produce real time results, while imposing low memory requirements.